Task delegation and cooperation for automated assistants

ABSTRACT

The present disclosure provides task delegation and cooperation for automated assistants. In one example, a method performed by a processing system that is in communication with a plurality of automated assistants includes receiving, a query indicating a task with which a user desires assistance, parsing the task into a plurality of sub-tasks including a first sub-task and a second sub-task, identifying a first automated assistant of the plurality of automated assistants that is capable of performing the first sub-task and a second automated assistant of the plurality of automated assistants that is capable of performing the second sub-task, and formulating a proposal for responding to the query, wherein the proposal comprises a sequence in which the plurality of sub-tasks is to be performed and a mapping that assigns the first sub-task to the first automated assistant and the second sub-task to the second automated assistant.

This application is a continuation of U.S. patent application Ser. No.16/595,645, filed Oct. 8, 2019, now U.S. Pat. No. 11,537,417, which isherein incorporated by reference in its entirety.

The present disclosure relates generally to artificial intelligence, andrelates more particularly to systems, non-transitory computer-readablemedia, and methods for providing task delegation and cooperation forautomated assistants.

BACKGROUND

Automated digital assistants (also sometimes referred to as “bots”) aresoftware applications that run automated tasks, often over the Internet.The automated tasks are typically simple and structurally repetitive,and can be performed by the automated digital assistants at a muchhigher rate than would be possible for a human to perform.

SUMMARY

The present disclosure provides task delegation and cooperation forautomated assistants. In one example, a method performed by a processingsystem that is in communication with a plurality of automated assistantsincludes receiving, a query indicating a task with which a user desiresassistance, parsing the task into a plurality of sub-tasks including afirst sub-task and a second sub-task, identifying a first automatedassistant of the plurality of automated assistants that is capable ofperforming the first sub-task and a second automated assistant of theplurality of automated assistants that is capable of performing thesecond sub-task, and formulating a proposal for responding to the query,wherein the proposal comprises a sequence in which the plurality ofsub-tasks is to be performed and a mapping that assigns the firstsub-task to the first automated assistant and the second sub-task to thesecond automated assistant.

In another example, a device includes a processor of a centralizedsupport center that is in communication with a plurality of automatedassistants and a computer-readable medium storing instructions which,when executed by the processor, cause the processor to performoperations. The operations include receiving, a query indicating a taskwith which a user desires assistance, parsing the task into a pluralityof sub-tasks including a first sub-task and a second sub-task,identifying a first automated assistant of the plurality of automatedassistants that is capable of performing the first sub-task and a secondautomated assistant of the plurality of automated assistants that iscapable of performing the second sub-task, and formulating a proposalfor responding to the query, wherein the proposal comprises a sequencein which the plurality of sub-tasks is to be performed and a mappingthat assigns the first sub-task to the first automated assistant and thesecond sub-task to the second automated assistant.

In another example, a non-transitory computer-readable storage mediumstores instructions which, when executed by a processor of a centralizedsupport center that is in communication with a plurality of automatedassistants, cause the processor to perform operations. The operationsinclude receiving, a query indicating a task with which a user desiresassistance, parsing the task into a plurality of sub-tasks including afirst sub-task and a second sub-task, identifying a first automatedassistant of the plurality of automated assistants that is capable ofperforming the first sub-task and a second automated assistant of theplurality of automated assistants that is capable of performing thesecond sub-task, and formulating a proposal for responding to the query,wherein the proposal comprises a sequence in which the plurality ofsub-tasks is to be performed and a mapping that assigns the firstsub-task to the first automated assistant and the second sub-task to thesecond automated assistant.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present disclosure can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 illustrates an example network related to the present disclosure;

FIG. 2 illustrates a flowchart of an example method for providing taskdelegation for automated assistants; and

FIG. 3 depicts a high-level block diagram of a computing devicespecifically programmed to perform the functions described herein.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION

In one example, the present disclosure provides task delegation andcooperation for automated assistants. As discussed above, automateddigital assistants are software applications that run automated tasks,often over the Internet. The automated tasks are typically simple andstructurally repetitive, and can be performed by the automatedassistants at a much higher rate than would be possible for a human toperform. However, the ability of conventional automated assistants tocooperate on tasks that may involve operations performed by multipledifferent automated assistants is limited. For instance, some automatedassistants are programmed to respond to a limited set of predefinedqueries, and cannot respond to queries outside of that limited set.

Other automated assistants may be capable of addressing one or moreportions of more complex queries, but may lack the mechanisms toeffectively resolve conflicts with other automated assistants. Forinstance, two different automated assistants (which may be provided bythe same vendor or may be provided by two different vendors) may becarrying out two different operations in the same temporal workspace.However, it may not be desirable for the operations of one automatedassistant to be started until the operations of the other automatedassistant have been completed, or at least performed to a specificpoint. As an example, a home renovation project may employ a firstautomated assistant to control and schedule work orders for hangingdrywall and a second automated assistant to control and schedule workorders for installing plumbing. Without a specific service levelagreement (SLA) negotiated between the vendors that provide theautomated assistants, however, it may be difficult to ensure that thedry wall is not hung until the after the plumbing is installed (e.g., aseach vendor, working independently of the other(s), may seekpreferential scheduling or approval statuses).

Examples of the present disclosure provide automated assistants that arecapable of breaking a complex query into a plurality of smallersub-tasks, which may be assigned to a plurality of different automatedassistants. Further examples of the present disclosure create afunctionality-based “exchange” via which the automated assistants thatare most appropriate for handling the sub-tasks may be identified. Stillfurther examples of the present disclosure evaluate the scheduling andassignment of the sub-tasks among the plurality of different automatedassistants in order to minimize conflicts between the differentautomated assistants. This allows the query to be resolved in an optimalmanner by delegating different sub-tasks to the automated assistantsthat are best-suited to perform the sub-tasks and by ensuring thatconflicts do not needlessly delay resolution of the query.

To better understand the present disclosure, FIG. 1 illustrates anexample network 100 related to the present disclosure. The network 100may be any type of communications network, such as, for example, atraditional circuit switched network (CS) (e.g., a public switchedtelephone network (PSTN)) or an Internet Protocol (IP) network (e.g., anIP Multimedia Subsystem (IMS) network, an asynchronous transfer mode(ATM) network, a wireless network, a cellular network (e.g., 2G, 3G, 4G,5G and the like), a long term evolution (LTE) network, and the like)related to the current disclosure. It should be noted that an IP networkis broadly defined as a network that uses Internet Protocol to exchangedata packets. Additional exemplary IP networks include Voice over IP(VoIP) networks, Service over IP (SoIP) networks, and the like.

In one example, the network 100 may comprise a core network 102. In oneexample, the core network 102 may combine core network components of acellular network with components of a triple play service network; wheretriple play services include telephone services, Internet services, andtelevision services to subscribers. For example, core network 102 mayfunctionally comprise a fixed mobile convergence (FMC) network, e.g., anIP Multimedia Subsystem (IMS) network. In addition, the core network 102may functionally comprise a telephony network, e.g., an InternetProtocol/Multi-Protocol Label Switching (IP/MPLS) backbone networkutilizing Session Initiation Protocol (SIP) for circuit-switched andVoice over Internet Protocol (VoIP) telephony services. The core network102 may also further comprise an Internet Service Provider (ISP)network. In one example, the core network 102 may include an applicationserver (AS) 104 and a database (DB) 106. Although a single AS 104 and asingle DB 106 are illustrated, it should be noted that any number ofapplication servers and databases may be deployed. Furthermore, for easeof illustration, various additional elements of core network 102 areomitted from FIG. 1 , including switches, routers, firewalls, webservers, and the like.

The core network 102 may be in communication with one or more wirelessaccess networks 120 and 122. Either or both of the access networks 120and 122 may include a radio access network implementing suchtechnologies as: global system for mobile communication (GSM), e.g., abase station subsystem (BSS), or IS-95, a universal mobiletelecommunications system (UMTS) network employing wideband codedivision multiple access (WCDMA), or a CDMA3000 network, among others.In other words, either or both of the access networks 120 and 122 maycomprise an access network in accordance with any “second generation”(2G), “third generation” (3G), “fourth generation” (4G), Long TermEvolution (LTE), or any other yet to be developed futurewireless/cellular network technology including “fifth generation” (5G)and further generations. The operator of core network 102 may provide adata service to subscribers via access networks 120 and 122. In oneexample, the access networks 120 and 122 may all be different types ofaccess networks, may all be the same type of access network, or someaccess networks may be the same type of access network and other may bedifferent types of access networks. The core network 102 and the accessnetworks 120 and 122 may be operated by different service providers, thesame service provider or a combination thereof.

In one example, the access network 120 may be in communication with oneor more user endpoint devices (UEs) 108 and 110, while the accessnetwork 122 may be in communication with one or more UEs 112 and 114. Inone example, the user endpoint devices 108, 110, 112, and 114 may be anytype of subscriber/customer endpoint device configured for wirelesscommunication such as a laptop computer, a Wi-Fi device, a PersonalDigital Assistant (PDA), a mobile phone, a smartphone, an email device,a computing tablet, a messaging device, a wearable “smart” device (e.g.,a smart watch or fitness tracker), a portable media device (e.g., an MP3player), a gaming console, a portable gaming device, a set top box, asmart television, and the like. In a further example, the user endpointdevices 108, 110, 112, and 114 may include smart home or IoT devices,such as smart doorbells, smart thermostats, smart lighting systems,smart locks, smart appliances, a smart in-vehicle assistant, or thelike. In addition, some of the UEs 108, 110, 112, and 114 may includesensors (e.g., imaging sensors, audio sensors, thermal sensors, pressuresensors, light sensors, smoke sensors, humidity sensors, motion sensors,gyroscopes, accelerometers, and/or the like) for detecting informationabout a surrounding environment. In one example, any one or more of theuser endpoint devices 108, 110, 112, and 114 may have both cellular andnon-cellular access capabilities and may further have wiredcommunication and networking capabilities (e.g., such as a desktopcomputer). It should be noted that although only four user endpointdevices are illustrated in FIG. 1 , any number of user endpoint devicesmay be deployed.

In one example, the endpoint devices 108 and 114 may have installedthereon an automated assistant (AA) 136 or 138, respectively. The AAs136 and 138 may comprise software applications that are programmed toallow the respective UEs 108 and 114 to perform tasks or sub-tasks,potentially under the direction of an individual or the AS 104. As such,the AAs 136 and 138 may perform all or part of the method discussedbelow in conjunction with FIG. 2 , which relates to delegating andcoordinating sub-tasks of query among a plurality of automatedassistants.

The access networks 120 and 122 may also be in communication with AAs140 and 142, respectively, that are not installed on UEs. These AAs 140and 142 may comprise, for instance, hardware devices such as robots,drones, or unmanned vehicles that are capable of performing tasks andsub-tasks under the direction of an individual or the AS 104 (e.g., thehardware devices can carry out the tasks or subtasks, possibly whilemeeting certain criteria such as time limits, cost restrictions, and/orthe like). The AAs 140 and 142 may also comprise software programs thatcontain executable code which, when executed by a processor orprocessing system, cause the processor or processing system to carry ourspecific operations. As such, the AAs 140 and 142 may perform all orpart of the method discussed below in conjunction with FIG. 2 , whichrelates to delegating and coordinating sub-tasks of query among aplurality of automated assistants.

In one example, the AAs 140 and 142 may have both cellular andnon-cellular access capabilities and may further have wiredcommunication and networking capabilities. In addition, some of the AAs140 and 142 may include sensors 144 (e.g., imaging sensors, audiosensors, thermal sensors, pressure sensors, light sensors, smokesensors, humidity sensors, motion sensors, gyroscopes, accelerometers,and/or the like) for detecting information about a surroundingenvironment.

The AAs 136, 138, 140, and 142 may communicate with each other as wellas with the AS 104 (e.g., via digital application programminginterfaces). The AAs 136, 138, 140, and 142 may also communicate withhumans (e.g., via input/output devices such as a display, a speaker, akeyboard, a camera, a microphone, a haptic device, or the like). The AAs136, 138, 140, and 142 may be owned and operated by individuals (e.g.,the same individuals who own and operate the UEs 108, 110, 112, and 114)or by service providers who make the AAs 136, 138, 140, and 142available for performing tasks on behalf of the individuals and/orservice providers. It should be noted that although four AAs areillustrated in FIG. 1 , any number of AAs may be deployed.

The AS 104 may comprise a computer specifically programmed to operate asa special purpose computer, as illustrated in FIG. 3 and discussedbelow. In one example, the AS 104 may perform the methods discussedbelow related to providing task delegation and coordination forautomated assistants. For instance, in one example, the AS 104 hosts anapplication that receives requests for assistance from UEs operated byusers, parses sub-tasks from the requests, and delegates the sub-tasksto a plurality of automated assistants. In this way, the AS 104 mayfunction as a centralized support center for orchestrating taskperformance. To this end, the AS 104 may include a parsing engine 124, asearch engine 126, a proposal formulation engine 128, and/or adelegation and monitoring engine 130.

In one example, the parsing engine 124 may comprise a device that parsesa user query into a plurality of sub-tasks. In one example, parsing ofthe query into sub-tasks may involve applying one or more techniques toparse a meaning from the query. For instance, natural languageunderstanding may be applied to determine that the query, “I am headinghome and hungry,” includes at least two sub-tasks: e.g., providingdirections from the user's current location to the user's home andmaking arrangements to provide food to the user. In addition, each ofthese sub-tasks may include its own set of sub-tasks, which may beparsed by the parsing engine or by an automated assistant to which thesub-task is assigned, as described in further detail below. By accessingthe user's preferences (which may be stored in a user profile, e.g., inthe DB 106), the parsing engine 124 may parse additional, impliedsub-tasks from the query, such as preparing the user's home for hisreturn.

In one example, the parsing engine 124 may correlate the sub-tasks withuser-specific information, which may be stored in the user profile, inorder to add nuance to the sub-tasks. For instance, certain terms usedby the user in the query may be subjective or may have particularmeaning to the user. As an example, as discussed above, the sub-task ofproviding directions to the user's home may involve determining theuser's current mode of transportation. The user's profile may indicatethe user's typical working hours and may further indicate that the userdrives to and from work. In this case, if the query, “I am heading homeand hungry” is received around the time of day that the user normallyleaves work, and the user's current location is determined to be his orher place of employment, then the parsing engine 124 may infer that theuser is currently traveling by car.

The parsing engine 124 may also identify sub-tasks or other aspects ofthe query that require more information. For instance, even where a userprofile may provide insight into the meanings of some query terms, theprofile may still be deficient with respect to other query terms. As anexample, the profile may not indicate the user's preferred types offood. In this case, the parsing engine 124 may request additionalinformation, e.g., directly from the user. Moreover, by requesting theadditional information from the user, the parsing engine 124 may learnuser preferences and patterns, which may be used to augment the userprofile so that the user profile is more useful when parsing futurequeries.

In one example, the search engine 126 may comprise a device that isconfigured to search a repository or exchange of automated assistants(e.g., part of DB 106) for automated assistants that may be capable ofperforming the sub-tasks identified by the parsing engine 124. Forinstance, the exchange may index a plurality of automated assistantsaccording to the features and functions of the automated assistants. Theindex may include tags or metadata referencing the features andfunctionalities of the automated assistants, e.g., such that the searchengine may match a keyword or keywords of a sub-task (e.g., “drivingdirections”) to the tags or metadata associated with an automatedassistant that is capable of performing the sub-task. In one example,the user profile may be consulted to guide the search of the exchange.For instance, automated assistants that have been used in the past torespond to queries from the user may be promoted or demoted forconsideration based on the user's feedback, as discussed in furtherdetail below.

In one example, the proposal formulation engine 128 may comprise adevice that is configured to formulate a proposal for responding to thequery, where the proposal arranges the plurality of sub-tasks in anordered list and maps the candidate automated assistants to thesub-tasks. For instance, the ordered list of sub-tasks may identify eachof the sub-tasks that should be performed in order to fully respond tothe query, as well as an order or sequence in which the sub-tasks shouldbe performed to avoid conflicts. As discussed above, it may not bedesirable to begin certain sub-tasks until certain other sub-tasks havebeen completed. In other cases, it may be possible to perform certainsub-tasks in parallel, or to interrupt certain sub-tasks to performcertain other sub-tasks. The mapping may assign, to each of thesub-tasks, at least one of the candidate automated assistants identifiedby the search engine 126. In one example, the proposal formulationengine 128 may formulate a plurality of different proposals anddetermine which proposal of the plurality of proposals best responds tothe query while also satisfying some constraint. The constraint mayinclude, for example, the monetary cost of the proposal (e.g., the costof utilizing the candidate automated assistants as well as the cost ofperforming the sub-tasks), the estimated amount of time needed for theproposal to fully respond to the query, or other constraints.

In one example, the delegation and monitoring engine 130 may comprise adevice configured to provide instructions to automated assistants thatare selected for carrying out the sub-tasks and to monitor the progressof the sub-tasks. Monitoring may be accomplished through the selectedautomated assistants sending updates to the delegation and monitoringengine 130 (e.g., periodically, when sub-tasks or specified percentagesof sub-tasks are completed, in response to inquiries from the processingsystem, etc.). Based on the monitoring (e.g., on the statuses of thesub-tasks), the delegation and monitoring engine 130 may modify orupdate the instructions previously sent to automated assistants, maysend new instructions instructing automated assistants to abandonpreviously delegated sub-tasks, and/or may send new instructions tonewly selected automated assistants for carrying out newly added and/orreassigned sub-tasks. In some examples, the delegation and monitoringengine 130 may also inform a first automated assistant when a sub-taskperformed by a second automated assistant, on which a sub-task to beperformed by the first automated assistant is dependent, has beensuccessfully completed.

In one example, the DB 106 stores information relating to automatedassistants that have registered with an exchange 132 via which theautomated assistants advertise their availability to provide certainfunctionalities. The exchange 132 may include, for each registeredautomated assistant, a primary and/or any secondary functionalities ofthe automated assistant (e.g., navigation services, ordering food from aspecific restaurant, controlling IoT devices, etc.). In addition, theexchange may be indexed by stored proposals or workflows that employedthe services of the automated assistant. In this case, if a currentquery shares similarities with a previously received query, thepreviously received query may be used as a search term or criterion toidentify the automated assistants that performed the sub-tasksassociated with the previous query. In a further example, the exchangemay provide feedback or ratings provided by users who have used theautomated assistant to perform sub-tasks.

In a further example, the DB 106 may also store a repository 134 offeedback, which may include feedback from users from whom past querieswere received. For instance, this feedback may indicate whether thequeries were responded to (e.g., sub-tasks completed) to the users'satisfaction or whether the users suggested improvements for respondingto the queries. The repository 134 of feedback may also include feedbackfrom automated assistants. For instance, this feedback may indicatewhether any problems were encountered in performing any sub-tasks and/orwhat actions were taken to resolve the problems.

In a further example, the DB 106 may include a repository 146 of userdata. In addition, the user data may include information about users'preferences, for example in the forms of profiles, query and feedbackhistories, or the like. User data may be used, for example, todisambiguate between potential meanings of queries, to identifypreferred automated assistants for carrying out sub-tasks, or to improvethe quality of response to user queries in other ways.

It should be noted that as used herein, the terms “configure” and“reconfigure” may refer to programming or loading a computing devicewith computer-readable/computer-executable instructions, code, and/orprograms, e.g., in a memory, which when executed by a processor of thecomputing device, may cause the computing device to perform variousfunctions. Such terms may also encompass providing variables, datavalues, tables, objects, or other data structures or the like which maycause a computer device executing computer-readable instructions, code,and/or programs to function differently depending upon the values of thevariables or other data structures that are provided.

Those skilled in the art will realize that the network 100 has beensimplified. For example, the network 100 may include other networkelements (not shown) such as border elements, routers, switches, policyservers, security devices, a content distribution network (CDN) and thelike. The network 100 may also be expanded by including additionalrobots, endpoint devices, sensors, access networks, network elements,application servers, etc. without altering the scope of the presentdisclosure.

To further aid in understanding the present disclosure, FIG. 2illustrates a flowchart of an example method 200 for providing taskdelegation for automated assistants. In one example, the method 200 maybe performed by an application server, e.g., AS 104 illustrated in FIG.1 . The application server may host a service that parses queries intosub-tasks, identifies automated assistants that are capable ofperforming the sub-tasks, and delegates the sub-tasks to the automatedassistants. However, in other examples, the method 200 may be performedby another device. For instance, any one of the automated assistants136, 138, 140, or 142 may be configured to perform all or part of themethod 200. As such, any references in the discussion of the method 200to components of the network illustrated in FIG. 1 are not intended tolimit the means by which the method 200 may be performed. By way ofexample, the method 200 is described as being performed by a processingsystem, such as the processing system 402 illustrated in FIG. 3 .

The method 200 begins in step 202. In step 204, the processing systemmay receive a query from a user, e.g., via an application executing onan end user device operated by the user. In one example, the query mayrelate to a task with which the user desires assistance. For instance,the user may speak or type the phrase, “I am heading home and hungry.”In further examples, other modalities can also be used for receivingqueries. For instance, the user may make a gesture that is associatedwith a specific query, may press a button on the end user device that isassociated with a specific query, may hold the end user device in anorientation (e.g., tilt, turn, shake, etc.) that is associated with aspecific query, or the like.

In step 206, the processing system may parse the query into a pluralityof sub-tasks. In one example, parsing of the query into sub-tasksinvolves applying one or more techniques to parse a meaning from thequery. For instance, natural language understanding may be applied todetermine that the query, “I am heading home and hungry,” includes atleast two sub-tasks: e.g., providing directions from the user's currentlocation to the user's home and making arrangements to provide food tothe user. In addition, each of these sub-tasks may include its own setof sub-tasks. For instance, providing directions may also includedetermining the user's current location and/or mode of transportation(e.g., car, train, bus, etc.). Making arrangements to provide food mayinvolve determining the current time of day (e.g., breakfast time, lunchtime, or dinner time), identifying the user's preferred types of food(e.g., Italian, spicy, vegetarian, etc.), and/or identifying businessesthat can provide the user's preferred types of food at the current timeof day (e.g., fast food, diner, takeout, etc.). Depending on the user'spreferences (which may be stored in a user profile), the query may implyadditional sub-tasks, such as preparing the user's home for his return(e.g., which may include further sub-tasks of turning on the lights,raising the temperature of the thermostat, and the like).

In one example, parsing the sub-tasks in accordance with step 206 mayinclude correlating the sub-tasks with user-specific information, whichmay be stored in the user profile, in order to add nuance to thesub-tasks. For instance, certain terms used by the user in the query maybe subjective or may have particular meaning to the user. As an example,as discussed above, the sub-task of providing directions to the user'shome may involve determining the user's current mode of transportation.The user's profile may indicate the user's typical working hours and mayfurther indicate that the user drives to and from work. In this case, ifthe query, “I am heading home and hungry” is received around the time ofday that the user normally leaves work, and the user's current locationis determined to be his or her place of employment, then the processingsystem may infer that the user is currently traveling by car. Similarly,if the user's profile indicates that the user is a vegetarian, theprocessing system may infer that the user does not want fast food. Theuser profile may store information that is explicitly provided by theuser (e.g., the user may enter information into a plurality ofpredefined fields intended to solicit the user's interests andpreferences) and/or information that is inferred by the processingsystem from previous queries or feedback from the user (e.g., theprocessing system may detect that the user often orders food from aspecific restaurant or requests navigation assistance to the samelocation at approximately the same time of day, Monday through Friday).Thus, the processing system may disambiguate between multiple potentialmeanings for a query based on information in the user profile, a contextof the user's present circumstances (e.g., at the time the query issubmitted), or other user-specific information.

Parsing the sub-tasks in accordance with step 206 may also includeidentifying sub-tasks that require more information before the sub-taskscan be effectively delegated. For instance, even where a user profilemay provide insight into the meanings of some query terms, the profilemay still be deficient with respect to other query terms. As an example,the profile may not indicate the user's preferred types of food. In thiscase, parsing the sub-tasks may involve requesting additionalinformation, e.g., directly from the user. For instance, the processingsystem may prompt the user to provide one or more preferred types offood in order to define the sub-task of making arrangements to providefood to the user. Moreover, by prompting the user for additionalinformation, the processing system may learn user preferences andpatterns, which may be used to augment the user profile so that the userprofile is more useful when parsing future queries.

In a further example, parsing the sub-tasks may include identifying anorder in which the sub-tasks should be performed. For instance, asdiscussed above, it may not be desirable to begin certain sub-tasksuntil certain other sub-tasks have been completed. In other cases, itmay be possible to perform certain sub-tasks in parallel, or tointerrupt certain sub-tasks to perform certain other sub-tasks. As anexample, where the sub-tasks include providing directions to a user'shome and making arrangements to provide food to the user, it may bepossible to make arrangements for the user to stop for takeout at arestaurant that is along the user's route home.

In step 208, the processing system may identify a set of (e.g., two ormore) candidate automated assistants that is capable of performing thesub-tasks. In one example, the processing system may search a repositoryor exchange (e.g., such as DB 106 of FIG. 1 ) that contains informationabout the capabilities of a plurality of automated assistants. Forinstance, the exchange may index a plurality of automated assistantsaccording to the features and functions of the automated assistants. Theindex may include tags or metadata referencing the features andfunctionalities of the automated assistants, e.g., such that theprocessing system may match a keyword or keywords of a sub-task (e.g.,“driving directions”) to the tags or metadata associated with anautomated assistant that is capable of performing the sub-task. The setof candidate automated assistants identified in step 208 may thuscomprise a subset of the automated assistants for which information isstored in the repository. For example, the set of automated assistantsmay include at least a first automated assistant that is capable ofperforming a first sub-task and a second automated assistant that iscapable of performing a second sub-task. In one example, a user profilemay be consulted to guide identification of the set of candidateautomated assistants. For instance, automated assistants that have beenused in the past to respond to queries from the user may be promoted ordemoted for consideration based on the user's feedback.

In one example, more than one candidate automated assistant may beidentified for at least one of the sub-tasks. That is, for any givensub-task, the processing system may identify multiple candidateautomated assistants, and further processing and analysis may beperformed (e.g., in accordance with the steps described below) in orderto select a candidate automated assistant that is best suited to performthe sub-task within the particular context of the query received in step204.

In step 210, the processing system may formulate a proposal forresponding to the query, where the proposal arranges the plurality ofsub-tasks in an ordered list and maps the candidate automated assistantsto the sub-tasks. In one example, the ordered list of sub-tasks may bederived from the parsing performed in step 206. For instance, theordered list of sub-tasks may identify each of the sub-tasks that shouldbe performed in order to fully respond to the query, as well as an orderor sequence in which the sub-tasks should be performed to avoidconflicts. In some examples, some sub-tasks may be performed in parallelto expedite the response to the query. The mapping may assign, to eachof the sub-tasks, at least one of the candidate automated assistantsidentified in step 208 (e.g., a first sub-task to a first candidateautomated assistant, a second sub-task to a second candidate automatedassistant, etc.). It should be noted, however, that the mapping will notnecessarily result in a one-to-one correspondence between sub-tasks andautomated assistants. For instance, multiple sub-tasks could bedelegated to a single automated assistant, or one sub-task could beshared by two or more automated assistants.

In one example, formulating the proposal may include evaluating aplurality of different proposals and determining which proposal of theplurality of proposals best responds to the query while also satisfyingsome constraint. The constraint may include, for example, the monetarycost of the proposal (e.g., the cost of utilizing the candidateautomated assistants as well as the cost of performing the sub-tasks),the estimated amount of time needed for the proposal to fully respond tothe query, or other constraints. In further examples, a plurality ofconstraints may be balanced against each other (where the relativepriorities of the various constraints may be specified by the userprofile). As an example, the sub-task of making arrangements to providefood to a user who is heading home could be fulfilled by arranging forthe user to stop on the way home for takeout or by arranging for food tobe delivered to the user's home sometime after the user is expected toarrive home. Arranging for delivery of food may be more expensive, butmay allow the user to sit down to a meal at home sooner than if the userstopped to pick up takeout.

In a further example still, candidate automated assistants may bid to beincluded in a proposal. A bid may include, for example, a cost (amonetary cost of use, a credit that is redeemable with a particularvendor, application, or service provider, etc.) associated with usingthe automated assistant in the proposal generally and/or a costassociated with using the automated assistant at a specific point in theproposal (e.g., for one of the first x sub-tasks). For example, if asub-task involves ordering food for delivery to the user's home,automated assistants provided by specific restaurants may includeincentives in the bids such as free delivery, discounts on food,loyalty/rewards programs, or other incentives. Any bids associated withthe candidate automated assistants may be considered as additionalconstraints on the formulation of a proposal.

In optional step 212, rather than autonomously selecting a proposal, theprocessing system may present a plurality of proposals (as formulated instep 210) to the user, e.g., by forwarding the plurality of proposals tothe application executing on the user's end user device. The pluralityof proposals may presented in an order that is ranked according to howwell the proposals satisfy some constraint (e.g., least expensive tomost expensive, least time consuming to most time consuming, etc.).

In step 214, the processing system may send instructions to a set ofselected automated assistants, where the instructions assign, to eachautomated assistant in the set of selected automated assistants, one ofthe sub-tasks according to a selected proposal (e.g., as formulated instep 210). For instance, a first instruction may assign a first sub-taskto a first automated assistant, a second instruction may assign a secondsub-task to a second automated assistant, and so on. In one example, theproposal may be selected by the processing system. In another example,the proposal may be selected by the user.

In one example, the instruction to a specific selected automatedassistant may define not only the sub-task that the selected automatedassistant is to perform, but also one or more constraints on when theselected automated assistant is to perform the sub-task (e.g., afterreceiving a signal indicating that another selected automated assistanthas completed another sub-task or that the user has arrived at apre-defined location, at a specific time on the clock or after aspecific amount of time has elapsed, etc.). For instance, if thesub-task involves ordering food for takeout from a restaurant that isalong the user's route home from work, a constraint associated with thesub-task may be to wait to order the food until an automated assistantthat is monitoring the user's travel indicates that the user is tenminutes away from the restaurant.

In some examples, step 214 may involve re-sending instructions to newlyselected automated assistants when an initially selected automatedassistant indicates that the task cannot be completed as instructed(e.g., due to malfunction of the initially selected automated assistant,demands of multiple users on the initially selected automated assistant,the initially selected automated assistant requiring clarification ofthe assigned sub-task, etc.). In another example, step 214 may involveabandoning or adding sub-tasks (and associated automated assistants) onthe fly, e.g., based on the progress of other sub-tasks. Abandoning oradding sub-tasks may also result in the modification of instructions toselected automated assistants assigned to other sub-tasks. For instance,on the user's way to pick up takeout at a restaurant, it may bedetermined that the user's car is running out of gas. A selectedautomated assistant that is tasked with navigation may be instructed, bythe processing system, to dynamically alter the user's route in order toinclude a stop at a gas station. In turn, another selected automatedassistant that is tasked with submitting the order for takeout may beinstructed, by the processing system, to delay submission of the order.

In step 216, the processing system may continuously monitor the progressof the proposal, including the completion of the sub-tasks by theselected automated assistants. Monitoring may be accomplished throughthe selected automated assistants sending updates to the processingsystem (e.g., periodically, when sub-tasks or specified percentages ofsub-tasks are completed, in response to inquiries from the processingsystem, etc.). As an example, the processing system may determine, basedon communication from an automated assistant that is tasked withordering takeout for the user to pick up on his way home, that the userhas picked up his food. The processing system may further determine,based on communication from an automated assistant that is tasked withnavigation, that the user is three miles away from home. The processingsystem may continue to communicate with the automated assistant that istasked with navigation until the processing system receives anindication that the user has arrived at home.

However, monitoring does not necessarily imply that the selectedautomated assistants are waiting on the processing system to provideinstructions or authorization to proceed with their sub-tasks. Forinstance, in one example, the selected automated assistants may becapable of communicating directly with each other. In this case, theinstructions sent to a selected automated assistant in step 214 mayinclude the identification and/or contact information (e.g., IP or mediaaccess control (MAC) address) of one or more other automated assistantswith which the selected automated assistant is to coordinate itsactions. Thus, an automated assistant that is tasked with orderingtakeout from a restaurant may communicate directly with an automatedassistant that is tasked with navigation to provide the location of andpick up time at the restaurant.

In step 218, the processing system may determine (e.g., based on themonitoring) whether the query has been satisfied. For instance, theprocessing system may determine whether all sub-tasks of the proposalhave been completed, where completion of all sub-tasks indicates thatthe query is satisfied. If the processing system concludes in step 218that the query has not been satisfied yet, then the method 200 mayoptionally return to step 214, and the processing system modify theinstructions sent to the selected automated assistants (e.g., bychanging an instruction already sent to a selected automated assistant,adding a new sub-task, abandoning a yet-to-be-completed sub-task, etc.)as necessary to expedite satisfaction of the query. If no modificationsto the instructions are deemed to be necessary, then the method maysimply return to step 216, and the processing system may continue tomonitor the progress of the proposal.

If, however, the processing system concludes in step 218 that the queryhas been satisfied, then the method may proceed to optional step 220. Inoptional step 220, the processing system may obtain feedback regardingthe satisfaction of the query. The feedback may be obtained directlyfrom the user, directly from one or more of the selected automatedassistants, or may be inferred by the processing system (e.g., based onwhether certain constraints on satisfaction of the query wereobjectively met).

In optional step 222, the processing system may update the user profileand/or other data structures in accordance with the feedback. Forinstance, if the feedback indicates that the user was dissatisfied withthe completion of a specific sub-task, then the processing system mayupdate the user profile to indicate that the automated assistant thatperformed the specific sub-task should not be used (or should be rankedlower than other candidate automated assistants) when responding tofuture queries from the user. Similarly, if the feedback indicates thatthe user was very satisfied with the completion of the specificsub-task, then the processing system may update the user profile toindicate that the automated assistant that performed the specificsub-task should be used (or should be ranked higher than other candidateautomated assistants) when responding to future queries from the user.

Optionally, all or part of the feedback received in step 220 could beshared with the vendors who provide the automated assistants. Forinstance, if a selected automated assistant was implicated in negativefeedback received from the user, the vendor who provides the selectedautomated assistant may want to know so that the vendor can makemodifications to improve the performance of the selected automatedassistant in the future. Similarly, a vendor may wish to know whatfactors went into a decision to include or not include a specificautomated assistant in a proposal, as this knowledge may help thevendors to modify the specific automated assistant and/or the manner inwhich the specific automated assistant may be marketed or presented(e.g., in the repository or elsewhere).

In a further example, where the feedback is positive, the processingsystem may store all or part of the proposal, e.g., as an example of howa specific query was successfully addressed. Thus, when future queriesare received that share some similarities with the specific query, allor part of the proposal that formed the response to the specific querymay be re-used to formulate proposals for responding to the futurequeries.

The method 200 may end in step 224.

The method 200 therefore simplifies the performance of tasks bycoordinating the actions of a plurality of automated assistants carryingout a plurality of sub-tasks that are part of a response to a largerquery. Continuous learning from prior experiences and/or human feedbackallows for conflicts between automated assistants to be minimized, sothat queries can be addressed more quickly and more efficiently.Additionally, the method 200 allows the actions of automated assistantsprovided by different vendors to be coordinated in a manner thatbenefits the user, without requiring an SLA between the vendors.

As discussed above, the method 200 may, in some examples, be carried outby one or more automated assistants rather than by a centralized server.For examples, a first automated assistant may parse complex queries toidentify sub-tasks, while another automated assistant may formulate aproposal for carrying out the sub-tasks.

Although examples of the disclosure are described within the context ofa user making an explicit request for performance of a task, it shouldbe noted that in some cases, a request for performance of a task maycome from an automated assistant, without explicit instruction from theuser. For instance, an automated assistant in a user's home may detectthe user complaining about the temperature in the home. The automatedassistant may communicate with another automated assistant, which may,in turn, adjust a thermostat in the home to make the home warmer orcooler.

FIG. 3 depicts a high-level block diagram of a computing devicespecifically programmed to perform the functions described herein. Forexample, any one or more components or devices illustrated in FIG. 1 ordescribed in connection with the method 200 may be implemented as thesystem 300. For instance, an automated assistant or an applicationserver could be implemented as illustrated in FIG. 3 .

As depicted in FIG. 3 , the system 300 comprises a hardware processorelement 302, a memory 304, a module 305 for providing task delegationand cooperation for automated assistants, and various input/output (I/O)devices 306.

The hardware processor 302 may comprise, for example, a microprocessor,a central processing unit (CPU), or the like. The memory 304 maycomprise, for example, random access memory (RAM), read only memory(ROM), a disk drive, an optical drive, a magnetic drive, and/or aUniversal Serial Bus (USB) drive. The module 305 for providing taskdelegation and cooperation for automated assistants may includecircuitry and/or logic for performing special purpose functions relatingto planning and/or performing tasks on behalf of a user. Theinput/output devices 306 may include, for example, a camera, a videocamera, storage devices (including but not limited to, a tape drive, afloppy drive, a hard disk drive or a compact disk drive), a receiver, atransmitter, a speaker, a microphone, a transducer, a display, a speechsynthesizer, a haptic device, a sensor, an output port, or a user inputdevice (such as a keyboard, a keypad, a mouse, and the like).

Although only one processor element is shown, it should be noted thatthe computer may employ a plurality of processor elements. Furthermore,although only one computer is shown in the Figure, if the method(s) asdiscussed above is implemented in a distributed or parallel manner for aparticular illustrative example, i.e., the steps of the above method(s)or the entire method(s) are implemented across multiple or parallelcomputers, then the computer of this Figure is intended to representeach of those multiple computers. Furthermore, one or more hardwareprocessors can be utilized in supporting a virtualized or sharedcomputing environment. The virtualized computing environment may supportone or more virtual machines representing computers, servers, or othercomputing devices. In such virtualized virtual machines, hardwarecomponents such as hardware processors and computer-readable storagedevices may be virtualized or logically represented.

It should be noted that the present disclosure can be implemented insoftware and/or in a combination of software and hardware, e.g., usingapplication specific integrated circuits (ASIC), a programmable logicarray (PLA), including a field-programmable gate array (FPGA), or astate machine deployed on a hardware device, a computer or any otherhardware equivalents, e.g., computer readable instructions pertaining tothe method(s) discussed above can be used to configure a hardwareprocessor to perform the steps, functions and/or operations of the abovedisclosed method(s). In one example, instructions and data for thepresent module or process 305 for providing task delegation andcooperation for automated assistants (e.g., a software programcomprising computer-executable instructions) can be loaded into memory304 and executed by hardware processor element 302 to implement thesteps, functions or operations as discussed above in connection with theexample method 200. Furthermore, when a hardware processor executesinstructions to perform “operations,” this could include the hardwareprocessor performing the operations directly and/or facilitating,directing, or cooperating with another hardware device or component(e.g., a co-processor and the like) to perform the operations.

The processor executing the computer readable or software instructionsrelating to the above described method(s) can be perceived as aprogrammed processor or a specialized processor. As such, the presentmodule 305 for providing task delegation and cooperation for automatedassistants (including associated data structures) of the presentdisclosure can be stored on a tangible or physical (broadlynon-transitory) computer-readable storage device or medium, e.g.,volatile memory, non-volatile memory, ROM memory, RAM memory, magneticor optical drive, device or diskette and the like. More specifically,the computer-readable storage device may comprise any physical devicesthat provide the ability to store information such as data and/orinstructions to be accessed by a processor or a computing device such asa computer or an application server.

While various examples have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a disclosed example shouldnot be limited by any of the above-described examples, but should bedefined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A method comprising: receiving, by a processingsystem that is in communication with a plurality of automatedassistants, a query indicating a task with which a user desiresassistance; parsing, by the processing system, the task into a pluralityof sub-tasks including a first sub-task and a second sub-task;identifying, by the processing system, a first automated assistant ofthe plurality of automated assistants that is capable of performing thefirst sub-task and a second automated assistant of the plurality ofautomated assistants that is capable of performing the second sub-task;and formulating, by the processing system, a proposal for responding tothe query, wherein the proposal comprises a sequence in which theplurality of sub-tasks is to be performed and a mapping that assigns thefirst sub-task to the first automated assistant and the second sub-taskto the second automated assistant.
 2. The method of claim 1, furthercomprising: sending, by the processing system, a first instruction tothe first automated assistant, wherein the first instruction instructsthe first automated assistant to perform the first sub-task; andsending, by the processing system, a second instruction to the secondautomated assistant, wherein the second instruction instructs the secondautomated assistant to perform the second sub-task.
 3. The method ofclaim 2, further comprising: monitoring, by the processing system, aprogress of the first sub-task by the first automated assistant and aprogress of the second sub-task by the second automated assistant. 4.The method of claim 3, further comprising: modifying, by the processingsystem, at least one of: the first instruction or the secondinstruction, in response to the monitoring.
 5. The method of claim 3,further comprising: receiving, by the processing system, user feedbackin response to a satisfaction of the query; and updating a profileassociated with the user, in response to the user feedback.
 6. Themethod of claim 3, further comprising: receiving, by the processingsystem, user feedback in response to a satisfaction of the query; andupdating a profile associated with the first automated assistant, inresponse to the user feedback.
 7. The method of claim 6, furthercomprising: sharing, by the processing system, the user feedback with aprovider of the first automated assistant.
 8. The method of claim 1,wherein the parsing is performed using natural language understanding.9. The method of claim 1, wherein the parsing includes disambiguatingbetween a plurality of potential meanings of the query, and wherein thedisambiguating is based on a profile of the user.
 10. The method ofclaim 1, wherein the parsing includes disambiguating between a pluralityof potential meanings of the query, and wherein the disambiguating isbased on a context of at least one present circumstance of the user. 11.The method of claim 1, wherein the identifying comprises: searching, bythe processing system, an exchange that stores data about respectivefunctionalities of the plurality of automated assistants.
 12. The methodof claim 11, wherein the searching comprises: matching a keywordassociated with the first sub-task to a tag associated with the firstautomated assistant in the exchange.
 13. The method of claim 1, whereinthe formulating comprises: formulating, by the processing system, aplurality of proposals for responding to the query, wherein the proposalis a selected one of the plurality of proposals that is determined torespond to the query while also satisfying a predefined constraint. 14.The method of claim 13, wherein the predefined constraint is a monetarycost of the proposal.
 15. The method of claim 13, wherein the predefinedconstraint is an estimated amount of time needed to carry out theproposal.
 16. The method of claim 1, wherein the plurality of automatedassistants is further in communication with each other.
 17. The methodof claim 1, wherein the sequence indicates that the first sub-task is tobe performed in parallel with the second sub-task.
 18. The method ofclaim 1, wherein the sequence indicates that the first sub-task is to beperformed before the second sub-task.
 19. A device comprising: aprocessor that is in communication with a plurality of automatedassistants; and a computer-readable medium storing instructions which,when executed by the processor, cause the processor to performoperations, the operations comprising: receiving a query indicating atask with which a user desires assistance; parsing the task into aplurality of sub-tasks including a first sub-task and a second sub-task;identifying a first automated assistant of the plurality of automatedassistants that is capable of performing the first sub-task and a secondautomated assistant of the plurality of automated assistants that iscapable of performing the second sub-task; and formulating a proposalfor responding to the query, wherein the proposal comprises a sequencein which the plurality of sub-tasks is to be performed and a mappingthat assigns the first sub-task to the first automated assistant and thesecond sub-task to the second automated assistant.
 20. A non-transitorycomputer-readable storage medium storing instructions which, whenexecuted by a processor that is in communication with a plurality ofautomated assistants, cause the processor to perform operations, theoperations comprising: receiving a query indicating a task with which auser desires assistance; parsing the task into a plurality of sub-tasksincluding a first sub-task and a second sub-task; identifying a firstautomated assistant of the plurality of automated assistants that iscapable of performing the first sub-task and a second automatedassistant of the plurality of automated assistants that is capable ofperforming the second sub-task; and formulating a proposal forresponding to the query, wherein the proposal comprises a sequence inwhich the plurality of sub-tasks is to be performed and a mapping thatassigns the first sub-task to the first automated assistant and thesecond sub-task to the second automated assistant.